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THE STATUS OF RELIABILITY ASSESSMENT 
FOR UNMANNED SATELLITES* 

William Wolman 

National Aeronautics & Space Administration 
Goddard Space Flight Center ^ 
Greenbelt, Maryland 


Summary “ 

Reliability Assessment techniques applied to Goddard’s 
unmanned space systems now have been in effect for about five 
years and it is useful to review the methodology used, its effec- 


tiveness and usefulness. Such a review encompasses not only the 
approach and methodology which has been used but also a compari- 
son of the assessment results prior to space flight with actual 



flight results. Major problem areas are the development of ade- 
quate mathematical models, the availability of information to 
estimate system, subsystem, component and part failure rates, 
the use of "correction" factors to compensate for the space 
environment and the proper interpretation of assessment results. 

The performance of assessments before actual flight is dis- 
cussed from the viewpoint of Establishing a plausible hypothesis 
which is subject to acceptance or rejection by the flight program. 
The question of interpreting a situation where assessment results 
show a relatively unreliable system but later flight results show 
success or the opposite is examined in the framework of scientific 
and rational models. 

* Presented at the Eleventh National Symposium on Reliability and 
Quality Control, Miami Beach, Florida, January 13, 1965. 
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Introduction 

The Goddard Space Flight Center of the National Aeronautics 
and Space Administration has managed thirty-four unmanned earth 
orbiting satellites which have accumulated in excess of twenty- 
five years of orbit life as of September, 1964. Included among 
these are scientific satellites such as Vanguard III and the 
Explorer series, weather satellites TIROS and Nimbus, interna- 
tional satellites Alouette and the United Kingdom (UK) series, 

\ 

observatories 0S0 and OGO, and passive and active communications 
systems such as Echo, Telstar, Relay, and Syncom. All these sat- 
ellites, with perhaps two exceptions, have provided us with vary- 
ing degrees of useful scientific and engineering information and 
knowledge. Individual lifetimes of satellites have varied from a 
few hours to 2 and 4 years of life. No two satellites launched 
were identical although the eight TIROS satellites launched were 
certainly similar — complexity in terms of number of parts range*; 
from two thousand for Explorer VII to a 5000 part magnitude for 
TIROS, and IMP (Explorer XVIII) with about 12,000 onto 30,000 — 
40,000 for OGO and Nimbus types. 

The purpose of my remarks is not to discuss any formal engi- 
neering and scientific information which has been gained but rather 
to talk about reliability assessment with particular emphasis on 
Goddard’s space experience. As you all know, we are dealing with 
a new discipline, one which does not have any extensive history of 
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over 10 years. There exist today some half dozen books on the sub- 
ject and to mention a few, they include Lloyd and Lipow, (1), 
Bazovsky, (2), Pieruschka, (3), and ARINC's "Reliability Engineer- 
ing", (4), handbook. As far as applications are concerned for the 
aerospace systems, the Proceedings of the National Symposium on 
Reliability and Quality Control represent the best available source 
of information. My remarks are expository in nature and represent 
my personal views on the subject and should not necessarily be 
taken as Goddard or NASA doctrine . 

A Rational Framework 

One of the first points I would like to discuss is the role 
of reliability assessment as part of a rational scientific frame- 
work of inference. By reliability assessment I am referring to 
the analytical processes which include the block or functional 
diagrams of a system, the mathematical model, the data used to 
estimate the parameters of the model, and the inferences made 
regarding the reliability of the system based on the results 
obtained. Reference (5) gives a more detailed discussion of dif- 
ferent types of reliability assessments and an example of a general 
mathematical model. Frequently, assessments for reliability are 
performed before a system is operated in its intended environment. 
For example, consider Goddard's Orbiting Astronomical Observatory 
(OAQ), prior to any space flight, it is desired to determine as 
much as possible about the reliability of the system on the basis 
of such factors as the design, parts information and test results. 


- 4 - 

Hence, in a general way, we can consider an assessment as an analy- 
sis which is performed prior to the performance of the actual expert 
ment of interest, namely, the space flight. For reliability assess- 
ments in the manner discussed, we are not dealing with any of the 
classical notions of statistical inference such as estimation or 
testing of hypothesis. The former of which deals with the estima- 
tion of parameters which are of interest on the basis of experi- 
mental results — for example, on the basis of several TIROS flights 
considered as a sample from a conceptual population, we can esti- 
mate the percent success, or the mean life of such a system. For 
hypothesis testing, first let us consider the classical concept. 
Webster defines a hypothesis as "A tentative theory or supposition 
provisionally adopted to explain certain facts and to guide in the 
investigation of others." A statistical hypothesis is a tentative 
statement regarding parameters, usually of an assumed distribution 
function. A hypothesis is either accepted or rejected on the basis 
of an experiment or sampling procedure. As an example, consider 
a sample from a manufacturing lot, we either accept or reject the 
lot for a stated quality level after observing the quality charac- 
teristics in the sample. The basic question is, therefore, how to 
relate a pre-experiment procedure such as reliability assessment 
to statistical inference concepts and procedures . There is one 

point which I would like to clarify. The data used to determine 

v 

the reliability of components, parts and subsystems, which are used 


as inputs in the reliability model for a system being assessed 
usually come from experimental sources , namely, laboratory or 
ground testing. However, the origin, test and environmental con- 
ditions of. such data are usually not known in sufficient detail 
and hence the "failure rates" can only be viewed as the best avail- 
able parametric values rather than random variables obeying sta- 
tistical laws. The assessed reliability of a system is not a 
statistical quantity but a fixed number obtained through an ana- 
lytical process which is not of known experimental origin. Hence, 
an assessment of the type outlined above serves most often only for 
the establishment of a plausible reliability hypothesis which is 
tested by a space flight program . 

Reliability Assessment and 
Statistical Inference 

A reliability assessment may be considered from the following 
viewpoint. In order to be more concrete let us assume that a reli- 
ability assessment for a satellite yields a value of .97 for a 
defined mission of assumed duration. Let us assume it is correct. 
Before the mission takes place, we are dealing with a "pre-data 
predictive" situation, in the sense discussed by Dempster, (6), 
before the event occurs, namely, predictlvely .97 can be con- 
sidered as a measure of the degree of certainty about an event, in 
this case the mission. After the mission has been completed, we 
are dealing with a "post-data predictive" situation which has 
resulted in one of two outcomes: 
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(1) If the mission was successful : "An event with 

probability .97 has occurred" 


(2) If it were unsuccessful : "An event with probability 

.03 has occurred" and we are surprised. 

It is important to realize that neither outcome "proves" or 

"disproves" the accuracy of the assessment. 

In more classical statistical inference theory we first 

establish the hypothesis (H): 

H: R - .97 

namely, the reliability (R) of the system is hypothesized to be .97, 
Under case (1) above, "success", we accept H and under case (2) we 
reject H. The single test would therefore have a power function 
as shown on Chart 1. 


CHART 1 


Probability 1 


Rejecting 



Reliability (R) 


.97 1 
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We are dealing with a binomial sampling situation with 
a sample size of n ** 1, and under success we accept H and 
under failure we reject H. Again, nothing can be proven 
regarding the absolute value of the reliability of the sys- 
tem tested. Furthermore , if we should reject H, i.e. R ■ .97, 
then we reject the assessment value and not the general method- 
ology underlying the assessment. In fact, we are rejecting some 
aspect of the particular assessment application. Going one step 
further, we can say that under rejection, the particular model 
implied by the assessment value is not consistent with observed 
fact. This situation is not uncommon in scientific investigations 
Naturally, if we continually reject H as additional systems are . 
flight tested, the more we would question the assessment model. 
This is because, as more experimental evidence is accumulated, 
the larger becomes the statistical power of the inference pro- 
cedure, and the more confidence we would have in the results indi 
cated. 

I would like to conclude this aspect of my discussion with 
two quotations. One is from a paper by a British medical re- 
searcher, (7), which is particularly pertinent and is as follows: 
"We have, however, underlined how misleading, and potentially dis 
astrous, it can be if a mathematical model is applied to a real 
situation which it does not truly represent. All mathematical 
models must simplify: that is their strength. They may, in 



over-simplifying, distort: that is their danger." The other 

quotation is made by two mathematical physicists, (8), "One often 
sees a statement that some result has been vigorously proved, 
unaccompanied by any verification that the conditions postulated 
in the proof are satisfied in the actual problem — and very often 
they are not. This misuse of mathematics is to be found in most 
branches of Science," and if I may add in many reliability assess- 
ment studies . 

Goddard Reliability Assessments 

Reliability assessment techniques have been used extensively 
on Goddard managed programs. There have been essentially two types 
of assessment operations. One of these stems from the policy by 
NASA to employ independent assessment contractors for major sys- 
tems. These contractors are non-hardware organizations, which 
under NASA direction perform reliability assessments on specified 
systems. Reliability contractors are selected on a competitive 
basis from a list of qualified bidders. The list on the following 
page represents the contractors which have or are performing reli- 
ability assessments for Goddard managed systems. 

In addition to performing reliability assessments, the con- 
tractors have supported the Goddard project managers, and in the 
case of CNES, the French project manager, in the performance of 
other reliability tasks such as failure mode analyses, test monitor 
ing, and parts application analyses. The role of the reliability 
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SUMMARY OF RELIABILITY CONTRACTS FOR GODDARD PROJECTS 


Space System 

OAO 

OGO 

Nimbus 

Advanced 
Syncom & 
ATS 

AOSO 

IMP D&E 

FR-1 


Independent Reliability Contractor 
Booz -Allen Applied Research 
Planning Research Corporation 
Operations Research, Inc. 
Planning Research Corporation 

Operations Research, Inc. 

Bird Engineering Associates 
ARINC Research Corporation* 


* This is a French contract administered and directed by 
CNES, Centre National D'Etudes Spatiales, Bretigny, 
France, the French space agency. 


contractors is advisory in nature and they act as consultants to 
the NASA project manager. The use of independent assessment con- 
tractors has usually been restricted to major projects which have 
prime or major subsystem contractors. Reliability assessments of 
a more minor nature on subsystems and assemblies are sometimes 
handled on a Goddard in-house basis. 

The other type of reliability assessment used on Goddard 
managed systems is that which is performed by the contractors 
themselves. I think that it is accurate to state that there has 
been some reliability assessment performed by virtually every space 
system contractor involved in Goddard managed projects. Some of 
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these are performed by contractor design organizations, others by 
reliability organizations, and others on a subcontract basis. 

Reliability assessments are also a requirement of most pro- 
posals for space hardware systems. Let me add that reliability 
assessments made for the same system concept by different organiza- 
tions are most difficult to compare as far as the absolute numbers 
are concerned, not only because the design approach may differ, 
but the mathematical models differ, failure rates differ and such 
factors as derating methods and "K-factors" are also not com- 
parable. The main basis for an evaluation must therefore rest on 
the approach taken and the concepts used to demonstrate capability 
and ingenuity rather than absolute values. I recall several 
instances where numerical reliability requirements were very ambi- 
tious, however, I recall no instance where a proposal did not meet 
these requirements — that is, on paper. 

A Comparison With Flight Results 

The most effective means of evaluating the adequacy of a reli- 
ability assessment is to compare assessment results with the actual 
operation of a space system in its intended environment. This is 
usually most difficult to do for space systems because experience 
to date, although substantial in terms of total accumulated orbit 
life, is not based on like systems. I would like to relate 
briefly a comparison between assessment and space operation which 
was performed for the first six TIROS meteorological satellites. 
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TABLE 1 

SUMMARY OF AVERAGE LIFE TIROS I THROUGH VI 


Function 

Observed 

Assessed 
Without "K" 
Factors 

Observed 

+ 

Assessed 

Assessed 
With "K" 
Factors 

Observed 

+ 

Assessed 



(Days) 

(Days) 


(Days) 


Direct TV 
At least one 

TV 

217 

230 

.94 

160 

1.36 

One specific 

TV 

140 

152 

.92 

105 

1.33 

Both TV 


64 

90 

.71 

60 

1.07 

Remote TV 
At least one 

TV 

215 

80 

2.69 

70 

3.07 

One specific TV 

131 

60 

2.18 

52 

2.52 

Both TV 


48 

43 

1.12 

35 

1.37 


A reliability contractor performed the reliability assessment and 
Goddard's TIROS project office furnished flight histories for 
TIROS I through VI. It should be pointed out that the TIROS satel- 
lites were not identical in design or manufacture. However, over- 
all differences were not considered of such a gross nature as to 
make summary statements meaningless. 

Table 1 summarizes the comparison between observed and assessed 
orbit life. The observed mean life figures in the table are based 
on averaging over the six TIROS flights. Twelve TV camera chains 
were used for "one specific TV" data as each TIROS contained two 
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cameras. The assessed values are given without and with "K- 
factors'* and are based on "graphical Integration" from curves and 

• j 

data in the reliability contractor's report. We can relate mean 
life to the area under a reliability curve with the following 
formula: „ 

Mean life - / R(t)dt, 

o 

where R(t) is the reliability or probability of exceeding mission 
time "t". Conclusions are that observed and assessed values are 
closer in one direction for the "Direct TV" than for the "Remote TV” 
and also that the comparison is better without "K-f actors". It is 
also interesting to note that for the observed life figures, the 
life for the redundant system, namely, "at least one TV" is not too 
much different from the expected 3/2 mean life theoretical value 
for one specific TV which is based on exponential theory. The 
above represents one specific comparison and does show that assess- 
ment results, using essentially standard techniques, are not grossly 
inconsistent with actual operation. In general, based on our 
limited experience to date, it can be concluded that carefully per- 
formed assessments can yield results which are not inconsistent 
with later experimental verification. I hasten to add though, that 
especially on individual flights, the discussion in the first part 
of these remarks is applicable, and that present assessment tech- 
niques are inadequate to predict operational results with suf- 


ficient confidence. 
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Conclusions and Recommendations 

The following conclusions and recommendations can be made: 

1. Reliability Assessment is a useful technique 
especially in the evaluation of alternative design 
approaches, the pinpointing of areas of unreli- 
ability, and to give management an order of magni- 
tude of the reliability of a system. 

2. Assessment results should be used very cautiously 
to predict operational reliability before flight 
history has been obtained. 

3. Much remains to be done in developing techniques for 
the analytical representation of complex systems as 
mathematical models. 

4. Data inputs such as part failure rates used in sys- 
tem assessments are often inadequate due to lack of 
test knowledge, effect of interaction and unknown 
environment. The same can be said for "K-f actors” . 

Summary Remarks and Acknowledgments 

My remarks should not be construed as an indictment of reli- 
ability assessment , on the contrary let me repeat as I have stated 
previously, (9), namely, there exists at present no other means of 
evaluating the reliability of a highly complex system before end 
use by using a rational approach and a quantitative basis than by 
using an approach at least similar in concept to that used for 
reliability assessment. History has shown that scientific and 
emotional factors are sometimes difficult to separate, one only 
has to review the work and life of Galileo and Copernicus and 
their problems with the science of astronomy. I am -not making 

t 

technical comparisons between the astronomer's woes and those of 


i 


reliability analysts, however, reliability assessments seem to 
bring out strong feelings pro and con, and these are often more 
concerned with the possible consequences of assessment interpre- 
tations rather than the technology involved in assessments per se. 

I would like to acknowledge the work of my colleague, Mr. 
Eugene Hixson, for his contributions in the TIROS study and for 
accumulating life history data for Goddard managed satellites. 

In addition, for the help of my other colleagues at Goddard, and 
numerous industry and government personnel who enabled me to make 
the above remarks, I wish to express my appreciation. 
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